Effective Quantization Approaches for Recurrent Neural Networks
نویسندگان
چکیده
Deep learning, Recurrent Neural Networks (RNN) in particular have shown superior accuracy in a large variety of tasks including machine translation, language understanding, and movie frames generation. However, these deep learning approaches are very expensive in terms of computation. In most cases, Graphic Processing Units (GPUs) are in used for large scale implementations. Meanwhile, energy efficient RNN approaches are proposed for deploying solutions on special purpose hardware including Field Programming Gate Arrays (FPGAs) and mobile platforms. In this paper, we propose an effective quantization approach for Recurrent Neural Networks (RNN) techniques including Long Short Term Memory (LSTM), Gated Recurrent Units (GRU), and Convolutional Long Short Term Memory (ConvLSTM). We have implemented different quantization methods including Binary Connect {-1, 1}, Ternary Connect {-1, 0, 1}, and Quaternary Connect {-1, -0.5, 0.5, 1}. These proposed approaches are evaluated on different datasets for sentiment analysis on IMDB and video frame predictions on the moving MNIST dataset. The experimental results are compared against the full precision versions of the LSTM, GRU, and ConvLSTM. They show promising results for both sentiment analysis and video frame prediction. Keywords— Deep Learning, Recurrent Neural Networks (RNN), LSTM, GRU, ConvLSTM, and Quantization.
منابع مشابه
کاهش رنگ تصاویر با شبکههای عصبی خودسامانده چندمرحلهای و ویژگیهای افزونه
Reducing the number of colors in an image while preserving its quality, is of importance in many applications such as image analysis and compression. It also decreases memory and transmission bandwidth requirements. Moreover, classification of image colors is applicable in image segmentation and object detection and separation, as well as producing pseudo-color images. In this paper, the Kohene...
متن کاملQuantized Memory-Augmented Neural Networks
Memory-augmented neural networks (MANNs) refer to a class of neural network models equipped with external memory (such as neural Turing machines and memory networks). These neural networks outperform conventional recurrent neural networks (RNNs) in terms of learning long-term dependency, allowing them to solve intriguing AI tasks that would otherwise be hard to address. This paper concerns the ...
متن کاملEffective Quantization Methods for Recurrent Neural Networks
Reducing bit-widths of weights, activations, and gradients of a Neural Network can shrink its storage size and memory usage, and also allow for faster training and inference by exploiting bitwise operations. However, previous attempts for quantization of RNNs show considerable performance degradation when using low bit-width weights and activations. In this paper, we propose methods to quantize...
متن کاملAn Integrated Neural Network Structure for Recognizing Autocorrelated and Trending Processes
Data sets collected from industrial processes may have both a particular type of trend and correlation among adjacent observations (autocorrelation). In the present paper, an integrated neural network structure is used to recognize trend stationary first order autoregressive (trend AR(1)) process. The proposed integrated structure operates as follows. (i) First a combined neural network structu...
متن کاملEfficient Short-Term Electricity Load Forecasting Using Recurrent Neural Networks
Short term load forecasting (STLF) plays an important role in the economic and reliable operation ofpower systems. Electric load demand has a complex profile with many multivariable and nonlineardependencies. In this study, recurrent neural network (RNN) architecture is presented for STLF. Theproposed model is capable of forecasting next 24-hour load profile. The main feature in this networkis ...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
برای دانلود متن کامل این مقاله و بیش از 32 میلیون مقاله دیگر ابتدا ثبت نام کنید
ثبت ناماگر عضو سایت هستید لطفا وارد حساب کاربری خود شوید
ورودعنوان ژورنال:
- CoRR
دوره abs/1802.02615 شماره
صفحات -
تاریخ انتشار 2018